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Preface 


The  System  Analysis  and  Studies  (SAS)  Panel,  which  is  one  of  the  technical 
panels  within  NATO’s  Research  and  Technology  Organisation,  commissioned 
a  Task  Group  (TG-034,  SAS  Activity  087)  to  write  a  Guide  for  judgement- 
based  Operational  Analysis  (OA).  After  an  exploratory  phase  in  2009  which 
addressed,  at  the  Panel’s  request,  the  issues  surrounding  use  of  judgement  in 
Operational  Analysis,  the  TG  formally  started  its  work  Spring  2010  and 
finalised  the  code  at  the  end  of  201 1. 

The  defence  environment  is  complex  and  dynamic.  The  nature  of  modem 
conflict,  rapidly  changing  technology,  changed  attitudes  to  risk,  and  the  sheer 
diversity  of  actors  from  different  cultural  backgrounds  are  factors  in  that 
environment.  Defence  decision  makers  are  confronted  with  an  increasing 
operational  complexity  which  has  strategic  implications.  Purely  mathematical 
methods  of  analysis  may  not  be  helpful  in  these  circumstances;  increasing  use 
needs  to  be  made  of  methods  which  rely  upon  the  structured  elicitation  and 
synthesis  of  judgements. 

Judgement -based  OA  (called  ‘soft’  OA  in  the  academic  world  to  contrast  with 
‘hard’,  mathematics-based  OA)  is  increasingly  used  to  support  defence  and 
security  decision  making  both  at  national  and  NATO  levels.  Such  decisions 
need  to  be  defensible  when  subject  to  scrutiny  and  decision  makers  must  have 
confidence  that  the  material  presented  to  them  is  the  best  available  so  that  the 
decision  risk  is  contained.  However,  judgement-based  analysis  cannot  be 
subjected  to  conventional  tests  of  mathematical  rigour,  so  an  alternative  strategy 
is  needed. 

This  volume  is  directed  to  the  clients  of  such  judgement-based  OA  studies. 
These  include  decision  makers,  study  sponsors,  end  users  and  other  stakeholders. 
Its  purpose  is  to: 

•  Create  an  understanding  of  what  judgement-based  OA  is,  and  what  it 
can  offer; 

•  Identify  the  requirements  for  the  client  group  in  sponsoring  and  guiding 
judgement-based  OA  studies;  and 
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•  Show  how  a  judgement-based  OA  study  is  carried  out  in  order  to 
maximise  the  validity,  credibility  and  acceptance  of  the  study  and  its 
outcomes. 

The  analyst-oriented  volume  of  the  Guide  (“Code  of  Best  Practice  for  ‘Soft’ 
Operational  Analysis”,  the  CoBP  itself)  describes  the  overall  study  methodology, 
the  study  process,  the  ‘actors’  involved  and  their  roles  and  responsibilities, 
the  achievement  of  validity,  credibility  and  acceptance,  and  the  communication 
with  the  client.  The  TG  proposes  that  its  work  be  complemented  by  an  education 
program  to  introduce  the  opportunities  offered  by  judgement-based  OA  to 
decision-making  bodies  within  NATO  and  Partner  Nations,  and  to  show  how 
to  make  best  use  of  it.  It  is  expected  that  once  published,  the  CoBP  will  be 
reviewed  and  revised  in  the  light  of  experience  in  practice.  A  third  volume  is  a 
brief  summarising  brochure  for  (high-level,  ‘executive’)  decision  makers 
explaining  key  aspects. 

The  adoption  of  the  Guide  is  expected  to  increase  significantly  the  acceptance 
of  judgement-based  studies  within  the  military  and  defence-oriented  operational 
analyst  communities.  This  will,  in  turn,  be  beneficial  to  the  quality  of  defence 
decision  making  through  the  enhancement  of  the  versatility  of  OA  support, 
to  both  operations  and  in  longer  term  support  of  strategy  and  defence  planning. 

The  analyst-oriented  volume  describes  the  TG’s  working  procedure,  and 
provides  additional  background  information  and  account.  The  TG  has  restricted 
itself  in  referencing  the  material  in  the  main  text  of  the  analyst  and  client- 
oriented  volumes  in  order  to  not  distract  the  reader  too  much  from  the  content. 
By  its  nature,  the  Guide  is  a  work  of  review  and  representation  of  relevant 
ideas.  References  are  given  where  appropriate  and  are  not  designed  to  be 
exhaustive,  or  even  comprehensive.  The  analyst-oriented  volume  contains  lists 
of  references;  the  client-oriented  volume  only  to  a  few  key  ideas.  The  last  page 
of  this  volume  lists  some  references  recommended  to  interested  clients. 

The  TG  acknowledges  with  gratitude  the  contribution  by  reviewers  from  both 
academia  and  defence  organisations  of  drafts  of  the  Guide.  Reviewers  of  this 
client-oriented  volume  were:  Dr.  R.A  Forder  (formerly  at  Defence  Science  & 
Technology  Laboratory,  United  Kingdom);  Lt.Col.  J-H.  Pay  (Norwegian 
Defence  Research  Establishment  FFI,  Norway);  Cpt.Cdr.  F.S.  Ordean  and 
Lt.Col.  I.  Psomas  (Joint  Assessment  Branch,  Joint  Force  Command  Brunssum, 
NATO). 
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Executive  Summary 


Judgement  in  different  guises  has  been  used  by  military  staff  whenever 
assessing  problematic  situations  and  making  decisions.  NATO  practitioners 
have  determined  that  approaches  within  Operational  Analysis  (OA)  that  are 
predominantly  based  on  human  judgement,  are  an  increasingly  critical 
capability  needed  to  support  defence  decision  making.  Wider  acceptance  and 
employment  of  judgement-based  OA  would  enable  the  defence  community  to 
enhance  its  capability  to  deal  with  complex,  high  priority  problem  issues  for 
which  this  type  of  analysis  is  particularly  suited. 

The  development  of  the  “NATO  Guide  for  Judgement-Based  Operational 
Analysis  in  Defence  Decision  Making”  was  commissioned  by  the  System 
Analysis  and  Studies  Panel  within  NATO’s  Research  and  Technology 
Organisation.  Its  purpose  is: 

•  To  create  an  understanding  of  what  judgement -based  OA  is; 

•  To  clarify  what  judgement-based  OA  can  do  to  help  address 
problematic  situations,  and  what  people  can  expect  from  it;  and 

•  To  provide  guidance  on  how  a  judgement-based  OA  study  should  be 
carried  out  to  maximise  validity,  credibility  and  acceptance  of  the 
study  and  its  outcomes. 

The  Guide  is  published  as  three  volumes:  an  analyst-oriented  document 
(the  “Code  of  Best  Practice  for  ‘Soft’  Operational  Analysis”)  setting  rules  of  the 
road  for  analysts,  this  client-oriented  document,  and  a  brief  summarising 
brochure  for  (high-level,  ‘executive’)  decision  makers  explaining  key  aspects. 
This  volume  aims  at  familiarising  decision  makers  and  their  staff  with  the 
potential  of  judgement-based  OA.  It  addresses  the  following  questions: 

•  What  is  judgement-based  OA? 

•  Which  problematic  situations  require  judgement-based  OA? 

•  How  does  judgement-based  OA  add  value? 

•  What  does  a  judgement-based  OA  study  look  like? 

•  What  is  expected  of  me,  the  client? 
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•  What  does  the  analyst  bring  to  achieve  validity,  credibility  and 
acceptance? 

•  How  can  a  Code  of  Best  Practice  protect  the  client  from  threats  to  the 
study? 
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Synthese 


Les  etats-majors  ont  utilise  le  jugement  sous  differents  aspects,  a  chaque 
fois  qu’ils  ont  evalue  des  situations  problematiques  et  pris  des  decisions. 
Les  professionnels  de  l’OTAN  ont  etabli  que  les  approches  utilisees  lors  de 
l’analyse  operationnelle  (AO),  qui  sont  principalement  basees  sur  le  jugement 
humain,  represented  une  capacite  critique  croissante  indispensable  au  soutien 
des  prises  de  decision  de  defense.  Une  meilleure  acceptation  et  une  meilleure 
utilisation  de  1’AO  permettraient  a  la  communaute  de  la  Defense  d’ameliorer  sa 
capacite  a  traiter  des  problemes  complexes  et  hautement  prioritaires  pour 
lesquels  ce  type  d’analyse  conviendrait  particulierement. 

L’elaboration  du  «  Guide  OTAN  pour  l’analyse  operationnelle  basee  sur  le 
jugement  dans  la  prise  de  decision  de  defense  »  a  ete  soutenue  par  la 
commission  sur  les  etudes  et  l’analyse  de  systemes  de  l’Organisation  pour  la 
Recherche  et  la  Technologie  de  l’OTAN.  Son  objectif  est  de  : 

•  Sensibiliser  a  l’AO  basee  sur  le  jugement ; 

•  Clarifier  la  maniere  dont  l’AO  basee  sur  le  jugement  peut  aider  a 
traiter  des  situations  problematiques,  et  ce  que  les  personnes  peuvent 
en  attendre ;  et 

•  Fournir  des  orientations  sur  la  maniere  dont  une  etude  par  AO  basee 
sur  le  jugement  doit  etre  menee  afm  d’optimiser  la  validite,  la 
credibility  et  l’acceptation  de  ladite  etude  et  de  ses  resultats. 

Le  guide  est  constitue  de  trois  ouvrages  :  un  document  d’analyste  (le  «  Code  de 
bonnes  pratiques  pour  une  analyse  operationnelle  ‘en  douceur’  »)  definissant 
les  regies  a  suivre  par  les  analystes,  le  present  document  oriente  «  client  »,  ainsi 
qu’un  court  resume  sous  forme  de  brochure  a  l’attention  des  decideurs  (de  haut 
niveau,  de  « l’executif »)  expliquant  les  principaux  aspects  de  cette  analyse. 
Le  present  ouvrage  a  pour  but  de  familiariser  les  decideurs  et  leurs  etats-majors 
avec  le  potentiel  de  l’AO  basee  sur  le  jugement.  II  aborde  les  questions 
suivantes  : 

•  Qu’est-ce  que  l’AO  basee  sur  le  jugement  ? 

•  Quelles  situations  problematiques  requierent  une  AO  basee  sur  le 
jugement  ? 
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En  quoi  l’AO  basee  sur  le  jugement  est-elle  source  de  valeur  ajoutee  ? 
A  quoi  ressemble  une  etude  par  AO  basee  sur  le  jugement  ? 

Qu’ attend- t-on  de  moi,  le  donneur  d’ordres  ? 

Que  realise  l’analyste  pour  apporter  a  l’etude  validite,  credibility  et 
acceptation  ? 

De  quelle  maniere  un  code  de  bonnes  pratiques  peut-il  rassurer  le 
donneur  d’ordres  contre  les  craintes  que  suscite  l’etude  ? 
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Chapter  1  -  WHAT  IS  JUDGEMENT-BASED  OA? 


The  science  of  assisting  decision  makers  to  find  and  compare  options  so  that 
capability  development  or  operational  decisions  may  be  made  is  known  to  the 
NATO  community  as  Operational  Analysis  (OA)  and  in  the  academic  world  as 
Operational  Research  or  Operations  Research.  No  distinction  of  application  is 
implied  by  the  substitution  of  ‘Analysis’  for  ‘Research’.  Although  definitions 
of  OA  abound,  below  are  two  examples  of  clear  and  concise  descriptions: 

•  ‘...the  application  of  scientific  methods  to  assist  executive  decision 
makers’  -  NATO  Code  of  Best  practice  for  CJTF  and  Component 
Commanders',  and 

•  ‘...the  discipline  of  applying  advanced  analytical  methods  to  help 
make  better  decisions’  or  more  simply  as  ‘the  science  of  better’  - 
UK  Operational  Research  Society. 

For  the  specific  purpose  of  this  document,  the  following  description  of  OA  is 
used  to  encapsulate  the  common  features: 

•  An  interdisciplinary  science  which  assists  better  decision  making 
through  identification  and  comparison  of  change  options  designed  to 
improve  problematic  situations. 

All  decisions  made  by  decision  makers  will  involve  judgement  to  some  extent. 
However,  this  is  distinct  from  the  notion  that,  in  some  cases,  the  material  used 
by  the  decision  makers  will  be  derived  from  a  methodical  process  that  uses 
human  judgement  and  predominantly  non-mathematical  models:  judgement- 
based  OA. 

Judgement-based  OA  is  used  to  inform  those  decisions  where  the  real  issues  at 
stake  and  the  factors  to  be  taken  into  account  are  incompletely  understood, 
where  many  and  possibly  conflicting  viewpoints  are  involved,  or  where  our 
understanding  of  the  actual  interactions  involved  is  limited.  Many  real-world 
issues  such  as  those  found  in  the  military  and  security  domains  can  be 
described  in  this  way  (see  the  case  study  on  Page  2  for  an  example).  The  term 
‘soft’  is  used  in  the  academic  community  to  denote  the  methods  used  in  such 
OA  studies  as  often  much  of  the  input  and  output  comprises  qualitative/ 
subjective  information. 
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WHAT  IS  JUDGEMENT-BASED  OA? 


Case  Study- A  Real-World  Example: 

Advice  to  the  Deployed  Commanders  in  East  Timor 


The  deployed  commanders  in  East  Timor  following  the  1 999  independence 
referendum  were  faced  with  a  challenging  issue  -  what  could  be  the  effect  of 
the  deployments  on  the  local  populations?  This  was  clearly  an  area  outside 
their  usual  expertise  and  experience  as  it  involved  non-military  considerations, 
an  unfamiliar  physical  and  human  environment  and  it  was  not  feasible  to 
reduce  all  the  issues  to  a  single  quantitative  measure  that  the  commander 
could  manage.  Instead  a  study  was  made  to  investigate  the  triggers  and 
causes  that  led  to  disruptive  events  in  that  country’s  400  year  history.  The 
findings  were  used  to  forecast  possible  events  that  could  occur  as  the 
population  reacted  to  the  changed  socio-political  environment.  While  the 
source  material  and  analytical  process  was  well  documented  it  was  primarily 
based  on  interpretation  and  judgement  of  material  collected  by  the  analyst. 
Similarly  the  use  made  of  the  material  was  to  guide  the  commander’s 
judgement  in  his  decisions  on  possible  courses  of  action. 


Cases  where  judgement-based  OA  is  used  to  understand  issues  described  above 
can  be  contrasted  to  cases  where  most  of  the  issues  of  a  problematic  situation 
are  known  and  there  is  a  mathematical  or  logical  process  to  derive  more 
quantitative  insights.  The  term  ‘hard’  is  often  used  here,  though  in  reality  there 
is  no  rigid  distinction  and  most  decisions  comprise  both  ‘hard’  and  ‘soft’ 
elements.  An  easy  way  to  understand  the  differences  in  emphasis  is  to  note  that 
‘hard’  OA  is  more  likely  to  be  based  on  verifiable  facts  and  numerical  data  via 
objective  modelling  whereas  ‘soft’  OA  is  predominantly  founded  on  judgement 
and  more  likely  to  lead  to  a  body  of  evidence  allowing  exploration  and 
comparison  of  possibilities  though  incorporation  of  subjectivity.  Many  real- 
world  issues  belong  to  the  latter  category;  particularly  high-level  questions 
relating  to  strategy,  capability  development  and  major  system  acquisitions. 

The  aim  of  judgement-based  OA  is  to  harness  and  exploit  the  latent  knowledge 
and  judgement  of  all  stakeholders  in  a  decision  in  a  coherent  and  auditable 
way.  Once  that  has  been  achieved  by  means  of  the  study  methods  addressed  by 
the  Guide,  the  findings  of  the  study  can  be  laid  before  the  responsible  executive 
decision  maker  as  evidence  to  assist  his  final  sovereign  judgement. 

The  output  of  a  ‘hard’  OA  study  will  always  have  a  quantitative  aspect,  though 
it  may  not  be  couched  in  terms  of  a  single  solution  as  sensitivity  and  ‘what  if 
issues  should  be  included.  In  other  cases,  competing  options  without  a  common 
reference  point  may  be  encountered.  For  instance,  decision  makers  may  address 
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an  issue  in  terms  of  cost  versus  reliability  where  the  data  has  been  presented 
based  on  strong  ‘hard’  and  verifiable  mathematical  OA  principles.  In  this 
case  the  two  measures  to  be  traded  (cost  and  reliability)  can  be  realistically 
optimised  (respectively  minimised  and  maximised)  as  single  solutions. 
However,  the  relative  value  of  these  two  measures  is  a  matter  of  debate 
between  the  decision  makers. 

The  output  of  a  judgement-based  OA  study  may  also  have  some  quantitative 
flavour,  depending  on  the  methods  used,  but  in  other  cases  it  may  be  a  body  of 
well-structured  evidence  that  allows  the  decision  maker  to  move  his  thinking 
forward.  In  the  extreme  cases  of  judgement -based  OA,  neither  the  options  nor 
the  means  of  comparing  them  will  be  known  in  the  first  instance. 
A  mathematical  solution  is  extremely  unlikely  as  is  the  notion  of  optimisation. 
Considerable  judgement  will  be  required  even  in  the  formulation  of  the  issues 
and  further  subjective  techniques  will  be  required  as  the  study  unfurls. 
Nevertheless,  decision  makers  still  need  to  have  material  on  which  to  base  their 
decisions  and  thus  the  analyst  and  client  community  must  have  a  sound 
strategy  to  deal  with  these  cases. 

In  fact,  there  is  a  continuous  spectrum  of  study  types  and  many  studies  are 
distinctly  hybrid  in  nature.  In  particular,  most  ‘hard’  OA  studies  benefit  from 
an  initial  judgement-based  problem-structuring  phase  to  bring  out  divergent 
stakeholder  perceptions,  clarify  issues  and  objectives,  identify  a  full  range  of 
options,  etc.,  while  some  inputs  to  ‘hard’  models  will  inevitably  fall  more 
into  the  category  of  judgement  but  this  does  not  mean  that  they  become  a 
judgement-based  study. 

For  the  client  of  the  analysis,  the  critical  issue  is  not  so  much  the  detail  of  the 
judgement-based  OA  methods  used  in  a  study  but  more  the  validity,  credibility 
and  acceptance  of  the  results  and  how  they  are  used  by  decision  makers.  Where 
there  is  measurable  data,  established  procedure  and,  critically,  verifiable 
mathematics,  the  material  presented  to  the  decision  makers  can  be  readily  and 
precisely  explained  and  defended.  In  judgement-based  studies  there  is  a  danger 
that  the  method  will  be  the  subject  to  as  much  scrutiny  by  the  decision  makers 
as  the  results.  This  may  be  particularly  apparent  where  those  involved  in  the 
decision  have  an  engineering,  mathematics  or  science  background  and  feel 
uncomfortable  with  the  style  of  judgement-based  OA.  Such  perceptions  may 
lead  to  delay  or  rejection  of  a  proposal  that  the  clients  intimately  involved  in 
the  study  are  proposing.  For  judgement -based  studies  an  analyst-oriented  Code 
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of  Best  Practice  (CoBP;  published  as  a  separate  volume  of  the  Guide)  is  needed 
to  ensure  that  the  best  possible  advice  was  presented  to  the  decision  makers. 

This  volume  of  the  Guide  aims  to  show  clients  how  to  commission,  guide  and 
use  judgement-based  studies  with  confidence,  resulting  in  less  decision  risk  to 
military  procedures,  organisations  and  acquisitions.  It  is  intended  as  a  reference 
to  provide  guidance  to  potential  clients.  This  volume  therefore  describes: 

•  The  nature  and  scope  of  judgement -based  OA; 

•  Potential  uses  of  judgement -based  OA; 

•  How  a  judgement -based  study  will  progress; 

•  The  requirements  on  and  obligations  of  the  client; 

•  The  expectations  and  needs  of  the  analyst;  and 

•  Potential  pitfalls  and  their  avoidance. 

Three  key  aspects  of  judgement-based  OA  should  be  noted: 

1)  It  enables  progress  to  be  made  for  some  otherwise  intractable  and 
complex  decisions. 

2)  It  involves  a  creative  journey  of  discovery  and  learning  that  can 
be  used  to  the  advantage  of  decision  makers. 

3)  The  inherent  uncertainty  of  complicated  decision  situations  that 
the  defence  sector  faces,  leads  the  client  for  judgement-based  OA 
to  what  are  perhaps  his  most  pressing  concerns  -  its  validity, 
credibility  and  acceptance.  Study  methods  must  therefore  be  well 
documented  to  withstand  scrutiny. 

Judgement-based  OA  is  in  many  ways  a  collaborative  adventure  involving 
subject-matter  experts,  clients,  analysts,  and  facilitators  of  workshops  with 
stakeholder  groups.  Good  practice  therefore  requires  that  all  are  aware  of  their 
roles  and  responsibilities.  This  volume  of  the  Guide  is  focused  on  issues 
regarding  general  aspects  of  judgement-based  OA,  particularly  from  the 
client’s  viewpoint,  and  on  how  that  contributes  to  rigorous  and  auditable 
quality  control  of  both  process  and  content. 

The  Guide  does  not  compare,  assess  or  recommend  individual  methodologies 
or  methods  and  techniques.  Many  textbooks  and  articles  that  attempt  to  do  that 
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already  exist  and  the  companion  analyst-oriented  volume  (see  its  Annex  B) 
gives  a  selective  list  and  a  brief  summary  of  methods  and  techniques. 


In  summary,  judgement-based  OA  is  characterised  by  its 
methodical  use  of  human  judgement  and  predominantly  non- 
mathematical  models.  It  is  used  where  there  are  real-world 
problematic  situations  that  cannot  be  solved  by  the  application 
of  predominantly  mathematics-based  sciences. 
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There  is  no  prescribed  single  way  to  identify  if  and  when  a  judgement-based 
analysis  is  needed.  If  there  is  uncertainty  or  disagreement  about  the  real  issues 
at  stake  and  the  factors  and  interactions  that  need  to  be  taken  into  account  or  if 
there  are  several  potential  ways  to  approach  the  issues,  then,  at  the  very  least, 
some  sort  of  initial,  judgement-based  analysis  to  structure  the  problem  is 
required.  If,  having  defined  the  issues  and  structured  the  problem,  one  then 
needs  to  analyse  interactions  where  understanding  of  the  real  world  is 
insufficient  to  construct  explicit,  mathematics-based  cause  and  effect  models, 
then  deeper  judgement -based  analysis  can  be  used  to  progress  the  study. 

The  check-list  in  Table  1  expands  on  these  basic  principles.  It  should  be  stressed 
that  ‘ticking’  several  of  the  items  on  the  list  should  not  be  seen  as  a  failing;  it  is 
just  a  recognition  of  reality.  As  will  be  seen,  the  list  can  be  used  to  advantage 
by  the  client. 
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Table  1:  Indicators  of  the  Need  for  a  Judgement-Based  OA  Study1. 


O  Not  much  is  initially  known  about  the  nature  of  the  problematic  situation 
and  its  boundaries  (what  matters  and  what  does  not). 

□  Not  much  is  initially  known  about  defining  elements  of  the  problematic 
situation  and  how  they  may  be  interrelated. 

□  Not  much  is  initially  known  about  who  the  stakeholders  are  and  in  what 
manner  they  may  be  directly  or  indirectly  affected,  their  viewpoints  and 
what  they  are  worried  about. 

□  Not  much  is  initially  known  about  goals,  objectives  and  measures  of 
effectiveness  or  merit  that  may  be  relevant. 

O  Not  much  is  initially  known  about  what  can  and  should  be  changed 
towards  improvement,  and  under  what  conditions  or  according  to  what 
criteria  a  change  will  be  regarded  as  an  improvement. 

□  Not  much  is  initially  known  about  the  data  needed,  its  relevance, 
availability  and  reliability. 

□  Not  much  is  initially  known  about  the  way  in  which  changes  in  context 
will  affect  the  problematic  situation,  its  improvement  and  the  study 
design  to  achieve  it. 

□  Power,  emotion,  politics  and  ethics  will  most  likely  come  into  play, 
but  not  much  is  initially  known  about  how  and  to  what  effect. 

□  Different  people  say  different  things  (or  express  different  views)  about 
the  same  issue. 

O  Not  much  is  known  about  key  interactions  of  human  cognition, 
beliefs  and  behaviour. 

□  Not  much  is  known  about  the  context  of  grand-scale  issues  that  have 
no  clear  end-points,  transcend  specific  domains,  and  have  significant 
political  or  public  policy  implications  (e.g.  counter-terrorism). 

□  Not  much  is  known  about  where  any  possible  resolution  will  most  likely 
have  side-effects  attached  to  them  that  are  undesirable  by 
stakeholders. 


Based  on:  Eden,  C.  and  Ackermann,  F.,  “Viewpoint  -  Where  next  for  problem  structuring  methods”, 
Journal  of  the  Operational  Research  Society,  57,  2006,  766-768;  Franco,  L.A.  and  Montibeller,  G., 
“Facilitated  modelling  in  operational  research”,  European  Journal  of  Operational  Research,  205,  2010, 
489-500;  Mingers,  J.,  “Soft  OR  comes  of  age  -  but  not  everywhere!”,  Omega,  39,  2011,  729-741; 
Pidd,  M.  (Ed.),  “Systems  modelling,  theory  and  practice”,  Wiley,  2004,  ISBN  0-470-86731-0; 
Pidd,  M.,  “Tools  for  thinking  -  Modelling  in  management  science”,  Wiley,  1996,  ISBN  0-471-96455-7; 
3rd  Edition  October  2009  (02010  ISBN  978-0-470-72142-1). 
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Examples  where  judgement-based  analysis  can  assist  decision  makers  include 
complex,  often  high-priority,  problem  issues  such  as: 

•  Broad  issues  that  are  at  an  early  stage  of  formulation  (e.g.  policy 
development); 

•  A  wide  range  of  concepts  to  replace  an  old  capability  being  examined, 
thus  avoiding  premature  adoption  of  a  particular  solution; 

•  Completely  new  areas  to  examine  (e.g.  to  address  newer  forms  of 
conflict  such  as  irregular/asymmetric  warfare); 

•  Acquisition  where  there  are  aspects  of  capability  which  may  need  to 
be  traded-off  (e.g.  mobility  versus  lethality);  or 

•  Planning  of  military-led  campaigns  that  need  to  be  identified 
and  examined  within  a  Whole  of  Government  or  comprehensive 
approach  context  where  ‘hearts  and  minds’  issues  are  a  major  factor 
(e.g.  counterinsurgency  or  Nation-building  in  failing  states). 

In  such  cases,  the  use  of  judgement-based  analysis  may  be  a  necessity  rather 
than  an  option  if  progress  is  to  be  made  ( see  key  aspect  1  in  Chapter  1). 
Any  attempt  to  treat  the  issue  as  something  that  can  be  logically  addressed, 
formulated  and  solved  in  a  purely  mathematically  way  will  be  fundamentally 
flawed,  and,  more  critically  for  the  client,  will  lead  to  rejection  or  delay  of  a 
decision. 

Cases  where  a  judgement-based  approach  is  not  required,  at  least  after  the 
problem-structuring  phase,  include  areas  where  clear  objectives  and  associated 
quantitative  measures  of  effectiveness  can  be  defined  and  where  our 
knowledge  base  allows  explicit,  objective  modelling  of  the  key  interactions. 
Examples  include:  inventory  management  and  scheduling  of  replacement  parts 
or  consumables,  maintenance  regimes,  workforce  planning  or  optimum  search 
patterns.  But  even  in  these  ‘clear-cut’  cases,  there  will  always,  at  some  level  of 
the  hierarchy  of  client  decision  makers  or  at  some  stage  of  the  study  process, 
be  different  interpretations  of  data,  results  or  findings;  there  will  be  experts 
providing  estimates  based  on  judgement;  and  there  will  be  group  discussions  to 
arrive  at  a  consensus  about,  for  example,  the  problem  formulation  to  begin 
with.  All  these  matters  require  approaches  that  are  based  on  judgement  and 
thus  fall  into  the  broad  category  of  judgement-based  OA. 
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i 


It  is  important  to  place  judgement-based  OA  in  context.  A  helpful  taxonomy  of 
describing  issues  is  to  use  the  established  terms  of  puzzles,  problems  and 
messes  (Table  2)2.  Together,  these  three  terns  describe  what  is  referred  to  as 
the  ‘problematic  situation’,  though  the  term  ‘problem’  is  often  used  to  denote 
both  a  specific  case  or  any  issue  examined  in  an  OA  study.  Short  descriptions 
of  these  areas  follow,  though  it  should  be  noted  that  these  terms  are  selected 
points  across  a  spectrum,  not  discrete  cases.  Nevertheless,  it  is  a  useful  concept 
and  many  studies  seem  to  deal  with  a  problematic  situation  that  falls  naturally 
into  one  column  or  another: 

•  ‘Puzzles’  -  These  are  cases  where:  (1)  The  issue  is  clear;  (2)  There  is 
probably  an  agreed  way  of  addressing  the  issue;  and  (3)  A  limited 
number  of  relevant,  well  defined  and  understood  measures  can  be 
evaluated  by  verifiable  mathematics.  In  general,  the  formulation  of  the 
issues  will  be  readily  agreed  by  the  client  and  analyst  and  the  results 
will  be  easy  to  validate. 

•  ‘Problems’  -  These  are  cases  where:  (1)  The  intent  of  the  study 
is  clear  though  there  might  be  several  ways  of  addressing  it; 
(2)  There  will  be  generally  established  ways  to  design  the  study; 
and  (3)  Quantitative  results  will  be  generated  using  reasonably  well- 
understood  measures  though  there  will  be  some  judgement  and 
therefore  subjectivity  involved  in  their  determination.  The  analyst  and 
client  will  be  able  to  agree  on  a  way  to  address  the  study  (‘agreeable’) 
though  the  client  may  claim  flexibility  as  to  the  actions  that  may  be 
taken  in  exploiting  its  results.  This  latter  point  is  not  necessarily  a  bad 
thing. 

•  ‘Messes’  -  These  are  cases  where:  (1)  There  is  a  perceived  need  to 
investigate  an  area  though  it  is  not  clear  what  the  issue  is  or  indeed 
how  to  even  describe  the  decision  context,  possibly  involving 
stakeholders  who  have  multiple  perspectives  and  disagree  about  what 
needs  to  be  done;  (2)  There  is  no  obvious  way  of  tackling  the  issue; 
and  (3)  Judgement  will  be  used  extensively  in  developing  and 
describing  possible  ways  forward  which  may  include  extensive 
qualitative  statements.  Involvement  of  the  many  stakeholders  means 
that  agreement  on  the  formulation  of  issues  is  not  guaranteed  and  the 
results  of  the  study  may  lead  to  wide  but  healthy  debate. 


2  What  follows  is  to  a  large  extent  based  on  M.  Pidd’s  work  and  references  therein  (see  footnote  1 ). 
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Table  2:  Characteristics  of  Puzzles,  Problems  and  Messes. 


Puzzle 

Problem 

Mess 

Description 

Well-defined 
issue  with  a 
specific  solution 
that  can  be 
worked  out. 

Well-defined  issue, 
but  with  no  single 
solution  or  approach. 

Complex  issue 
which  is  not  well 
defined. 

Formulation 
of  the  Issues 

Agreed  objectives 
achieved  through 
logical  analysis. 

Agreeable  after 
negotiation  between 
clients  and  analysts, 
and  input  by  other 
interested  parties. 

The  lack  of  clarity 
and  agreement  of 
the  issues 
themselves  will 
make  formulation 
disputed. 

Typical 

Advice 

Optimal  solution. 

Preferred  option 
based  on  ranking. 

Possible  courses 
of  action. 

Validity  of 
the  Advice 

Accepted  after 
scrutiny  of 
mathematics  and 
calculations. 

Even  though  the 
results  are  reached  by 
agreed  methods,  the 
conclusions  can  be 
disputed  thus  leading 
to  informed  debate 
about  the  actions  to 
take. 

The  advice  will  lead 
to  an  informed 
debate  about  the 
pros  and  cons  of 
several  possibilities 
for  the  way  ahead. 

The  need  to  use  judgement -based  OA  will  increase  towards  the  right-hand  side 
of  Table  2. 

Figure  1  shows  that  when  moving  from  puzzles  to  messes,  the  clear-cut  nature 
of  the  problematic  situation  will  diminish,  whilst  reliance  on  judgement  will 
increase. 
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clear-cut 
nature  of 
situation 


more  clear-cut 
situation  requiring  less 
judgement 


reliance  on  human 
judgement 


Figure  1:  A  Spectrum  of  Problematic  Situations  and  Their 
Clear-Cut  Nature  versus  Reliance  on  Human  Judgement. 


The  initial  design  of  a  judgement-based  approach  should  be  left  to  the  analyst 
though  the  client  will  need  to  understand  and  agree  the  way  ahead.  The  material 
in  this  volume  will  help  to  identify  the  likely  course  of  action  that  the  analysts 
will  take  and  thus  will  inform  initial  discussions,  provide  a  foretaste  of  the 
approach  to  be  taken  and  guide  client  actions  and  expectations.  The  companion 
analyst-oriented  volume  will  provide  greater  detail  of  how  the  study  may  be 
conducted. 

Considerable  value  may  be  added  by  adopting  elements  of  a  judgement -based 
approach,  even  for  a  study  which  may  appear  at  first  glance  to  be  puzzle-like. 
The  client  is  advised  to  use  the  check  list  given  in  Table  1  to  see  if  consideration 
of  any  of  the  points  provides  better  insights. 

Finally  it  should  be  noted  that  Chapter  1  already  pointed  out  that  there  is  a 
continuous  spectrum  of  study  types  and  many  studies  are  distinctly  hybrid  in 


RTO-SAS-087 


11 


WHICH  PROBLEMATIC  SITUATIONS 
REQUIRE  JUDGEMENT-BASED  OA? 


nature.  A  predominantly  judgement-based  approach  may  include  therefore 
some  ‘hard’  or  quantitative  elements.  This  is  referred  to  as  ‘complementarity’: 
‘soft’  and  ‘hard’  methods  and  techniques  complement  each  other.  It  is  often  the 
case  that  the  analyst  needs  to  dig  deep  into  a  problem  to,  for  example,  test 
feasibility  or  consequences  of  adopting  a  particular  option. 


In  summary,  a  judgement-based  OA  approach  should  be 
taken  when  the  issues  are  not  initially  clear  or  agreed,  where 
multiple  perspectives  across  stakeholders  have  to  be  dealt 
with,  or  there  is  no  obvious  and  clear  way  to  conduct  the 
study  owing  to  an  insufficient  understanding  of  the  real  world. 
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Judgement-based  OA  is  in  itself  conducted  as  a  creative  journey  of  discovery 
and  learning  ( see  key  aspect  2  in  Chapter  1).  This  results  from  the  needs 
imposed  by  the  nature  of  ‘problems’  and  ‘messes’  as  opposed  to  ‘puzzles’. 
Judgement-based  OA  uses  models  both  for  visualisation  of  the  issues  and  for 
manipulation  to  explore  possible  changes  or  evaluate  options.  Such  models 
may  include  the  beliefs,  insights  and  expectations  of  people  who  wish  to  use 
the  model  as  a  means  of  communication  and  stimulus  of  debate  and  reflection 
(i.e.  support  their  thinking)  to  create  or  improve  a  mutual  understanding  of 
each  other’s  viewpoints  and  positions.  This  creation  or  improvement  of  mutual 
understanding  is  part  of  the  learning  process. 

As  such,  judgement-based  OA  adds  value  through: 

•  An  improved  shared  understanding,  through  a  recognition  to  consider 
all  sides  of  the  issue; 

•  An  enhanced  sense  of  common  purpose; 

•  A  greater  commitment  to  a  general  way  forward; 

•  The  discovery  and  consideration  of  alternative  options; 

•  The  development  of  acceptable  solutions  or  ways  forward  to  improve 
the  problematic  situation; 

•  Harnessing  of  latent  knowledge  and  the  systematic  gathering  and 
analysis  of  information;  and 

•  The  use  of  an  approach  that  recognises  people’s  different  cognitive 
viewpoints  and  belief  systems. 

All  of  these  items  contribute  to  the  development  of  a  shared  common  picture  of 
the  issues,  even  if  there  is  disagreement  on  detail.  Thus  an  informed  climate  of 
debate  is  developed  (see  case  study  on  Page  14). 
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Case  Study:  Creative  and  Cooperative  Issue 
Exploration  -  The  Army  of  Tomorrow  Study 


A  Seminar  War  Game  (SWG)  on  the  ‘Army  of  Tomorrow”  examined  military 
operations  of  the  future  by  incorporating  both  military  and  civilian  (e.g.  police, 
diplomats,  aid  agencies,  scientists,  and  others)  judgements  and  perspectives. 
It  provided  a  means  to  ‘meld’  these  diverse  judgements  together  in  order  to 
produce  insights  into  future  operational  challenges  and  opportunities. 

The  judgement-based  OA  approach  provided  a  means  to  engage  stakeholders 
who  came  from  different  cultures  and  had  very  different  points  of  view. 


Key  elements  of  judgement-based  OA  that  must  be  justified  and  explained 
include: 

•  The  methodology  used  (defined  as  ‘the  particular  combination  of 
methods  or  techniques  used  in  the  study 

•  The  particular  methods  or  techniques  used  (each  is  ‘a  structured  set 
of  guidelines  or  rules  or  algorithms  to  achieve  some  clear  well-defined 
purpose '). 

•  Models  used  in  the  study  (each  is  ‘a  representation  of  (a  part  of) 
reality  as  seen  by  a  number  of  people  who  wish  to  use  the  model  to 
understand,  manage,  or  improve  that  reality '). 

In  this  context  we  refer  to  a  model  as  an  abstraction  of  reality  that  supports 
an  OA  study.  Use  of  models  lies  at  the  heart  of  all  types  of  OA,  but  they  may 
not  necessarily  involve  mathematics.  Famously,  they  may  even  not  be  right! 
(see  text  box  below). 


‘All  models  are  wrong!’ 


It  is  only  meaningful  to  debate  validity  of  a  method  at  the  point  of  application, 
i.e.  in  the  context  of  the  decision  problem  being  addressed. 

“All  models  are  wrong,  but  some  are  useful."1 


Types  of  judgement-based  OA  models  that  can  add  value  include  the  following: 

•  Rich  pictures  allowing  a  visual  representation  of  the  issues  on  a  single 

sheet  of  paper. 


1  G.E.P.  Box,  N.R.  Draper,  “Empirical  Model-Building  and  Response  Surfaces”,  Wiley,  1987. 
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•  Discrete-event  linked  diagrams  showing  the  stages  involved  in  a 
process  of  change  and  development. 

•  Influence  diagrams  showing  how  discrete  elements  in  a  system  are 
linked  (sometimes  as  cause-effect  relationships  in  ‘causal  maps’)  and 
the  trade-offs  that  may  be  required. 

•  Mind  maps  (or  ‘concept  maps’)  to  explore  concepts  and  to  identify 
and  classify  issues. 

•  SWOT  matrices  pairing  Strengths  and  Weaknesses  with  Opportunities 
and  Threats. 

•  Hierarchical  diagrams  of  concepts  and  their  relationships. 

•  Maps  of  benefits  and  their  mechanisms  of  generation. 

When  considering  the  types  of  problematic  situation  we  may  wish  to  address, 
the  differences  between  the  ‘puzzle’  and  ‘mess’  ends  of  the  spectrum  in  Table  2 
identify  benefits  that  can  be  exploited  using  a  judgement-based  approach 
(Table  3). 
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Table  3:  Benefits  of  Taking  a  Judgement-Based  Approach 
Based  on  the  Differences  between  Predominantly  Judgement- 
Based  and  Predominantly  Mathematics-Based  OA2. 


Differences:  ‘Mess’  vs.  ‘Puzzle’ 

Benefits  of  Judgement- 
Based  Approach 

Study 

Outcomes 

Explorative  and  qualitative 
leading  to  insights  and  learning 
rather  than  seeking  statistically 
valid  outputs. 

Allows  management  of  the 
way  forward  rather  than 
finding  the  solution. 

Study 

Purpose 

Must  be  discussed  and  reviewed 
after  much  analysis  rather  than 
deduced  through  problem 
analysis. 

Avoids  going  too  early  to  a 
solution. 

Problematic 

Situation 

Mental  construct  formulated  in 
terms  of  a  manageable  'problem' 
or  ‘mess’  rather  than  a  logical 
process  to  reach  a  solution. 

Encourages  longer-term 
thinking  and  the  ability  to 
reconcile  different  objectives, 
perspectives  and  values. 

Process  - 
Study 

Stages  emerge  as  appropriate 
instead  of  in  a  planned  and 
sequential  manner. 

Iterative  interpretation. 

Process  - 
People 

Clients  (and  other  stakeholders) 
contribute  by  participation  and 
informed  judgement  (sometimes 
affected  by  power  and  emotion), 
in  addition  to  (possibly  sparse) 
hard  data. 

Biases  are  identified  and 
accommodated,  with  clients 
identifying  creative  options. 

Methodology 

Predominantly  interpretative 
and  subjective  instead  of 
predominantly  factual  and 
objective. 

Allows  freedom  to  explore 
possibilities  through  an 
interpretative  approach. 

Models 

Representations  of  the 
problematic  part  of  the  perceived 
world  rather  than  the  real  world. 

Conceptual  visualisation. 

Data 

More  observer-dependent  and 
subjective  (affected  by  a  broader 
range  of  sources  of  uncertainty) 
than  observer-independent  and 
objective. 

Uncertainty  is  treated  in  a 
creative,  explorative  manner, 
allowing  'what  if  thinking. 

1  Partly  based  on  M.  Pidd’s  work  (references  listed  under  Chapter  2’s  footnote  1  on  Page  7). 
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Many  of  the  positive  aspects  mentioned  so  far  stem  from  the  need  in 
judgement-based  OA  for  a  divergent  discovery  phase  (Figure  2)  where  all 
aspects  of  the  issue  are  discussed  and  explored.  The  divergent  phase  enables 
stakeholders  to  minimise  the  risk  of  pre-conceived  notions  becoming  dominant 
before  the  ‘problem’  has  been  derived  from  the  ‘mess’.  This  should  be 
compared  with  the  case  for  more  structured  issues  where  there  will  be  steady 
convergence  from  the  start.  The  client  must  expect  and  be  comfortable  with 
this  initial  divergent  problem  structuring  stage  and  be  ready  to  realise  the  value 
it  might  add. 


Divergent  study 


Convergent  study 


Figure  2:  Comparison  of  the  Divergence  and  Convergence  in  Judgement- 
Based  (left)  and  Mathematics-Based  (right)  OA  Studies. 


In  summary,  judgement-based  OA  provides  value-adding  and 
explorative  methods  to  identify  possible  ways  to  tackle  ill-defined 
issues.  The  value  added  includes:  harnessing  of  latent 
knowledge,  the  involvement  of  stakeholders,  the  exploration 
and  formulation  of  the  problematic  situation,  and  the  creation 
of  a  model  of  that  situation  which  can  be  agreed  on  and 
exploited  to  inform  decisions. 
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Chapter  4  -  WHAT  DOES  A  JUDGEMENT- 
BASED  OA  STUDY  LOOK  LIKE? 


Although  judgement-based  OA  studies  will  follow  the  same  general  principles, 
none  will  be  alike  as  each  investigation  will  tend  to  be  unique.  The  variable 
factors  are,  typically:  the  subject  matter,  the  desired  outcome  and  the  dynamic 
relationships  of  the  combination  of  the  client  and  analyst.  In  general,  any  OA 
study  will  follow  the  formula  of: 

•  Identify  the  stakeholders; 

•  Achieve  consensus  on  the  issues; 

•  Describe  the  system  of  interest  to  the  stakeholders  and  benefits  they 
seek  from  it; 

•  Identify  the  system  measures  of  performance  and  benefit; 

•  Generate  change  options;  and 

•  Model  the  system  of  interest,  so  that  change  options  can  be  evaluated 
for  the  benefits  each  will  deliver. 

An  OA  study  follows  an  iterative  process  which  allows  the  study  team  (i.e.  the 
clients,  the  analysts  and  other  stakeholders)  to  work  collaboratively  towards  a 
developing  goal.  Decision  makers,  faced  with  a  need  to  formulate  a  plan  of 
action,  will  express  their  predicament  as  a  problematic  situation  to  which  some 
structure  should  be  given.  Analysts  are  asked  to  suggest  a  design  for  a  study  of 
the  problematic  situation;  the  design  evokes  methods,  models  and  data  in  an 
iterative,  and  hopefully  convergent,  programme  of  analysis  which  may  include 
objective  knowledge  from  the  worlds  of  science,  mathematics  and  engineering. 
This  process  can  be  described  through  a  four-phase  process1  of: 

•  Appreciation; 

•  Analysis; 

•  Assessment;  and 

•  Action. 


Based  on  Mingers,  J.  and  Brocklesby,  J.,  “Multimethodology:  Towards  a  framework  for  mixing 
methodologies”,  Omega  25,  1997,  489-509. 
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This  process  is  in  many  ways  akin  to  the  military  OODA  loop  process  of 
Observe,  Orient,  Decide  and  Act.  Figure  3  shows  the  iterative  process  of  a 
judgement-based  OA  study,  where  the  goal  of  the  study  may  not  have  been 
apparent  at  the  start  and  indeed  may  change  several  times  during  the  process. 
During  the  process  there  are  also  key  deliverables  that  have  to  be  agreed  upon 
by  clients  (i.e.  their  decision  makers). 


1  4  1  4 


Figure  3:  The  Iterative  Judgement-Based  OA  Study  Process. 


Within  these  guidelines  many  possible  paths  may  be  taken  for  any  given  study 
(Table  4),  involving  various  methodologies  or  methods,  all  of  which  will 
inform  the  client  and,  ultimately,  be  useful  to  a  decision  maker.  The  key 
aspects  are  the  continuing  dialogue  between  the  client  and  the  analysts  and 
iterative  revision  of  the  models  and  options.  The  four  phases  should  therefore 
be  used  in  an  iterative  and  complementary  manner.  While  Figure  3  is  also 
applicable  to  a  ‘hard’  OA  study,  the  degree  of  iteration,  reinterpretation  and 
revision  is  likely  to  be  much  higher  for  a  judgement-based  one  because  of  the 
inherent  uncertainty. 
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Table  4:  Typical  Elements  of  a  Judgement-Based  OA  Study. 

Appreciation: 

•  Hold  workshops  with  stakeholders  (possibly  including  subject-matter  experts)  to 
gather  their  views  of  the  problematic  situation. 

•  Interview  clients  (including  those  affected  by  the  study’s  outcome),  actors 
(participants)  and  owners  (those  with  power). 

•  Draw  rich  pictures  based  on  the  views  expressed  by  the  stakeholders  and  agree  a 
common  picture. 

•  Identify  the  vocabulary  needed  to  describe  the  system  of  interest  and  changes  to  it. 

Analysis: 

•  Identify  relationships  between  the  factors  of  relevance. 

•  Visually  structure  the  factors  in  a  diagram  or  some  other  type  of  (visual)  model. 

•  Analyse  the  map  and  determine  crucial  factors  for  focus. 

•  Propose  measures  to  characterise  the  benefits  desired  and  explore  some  plausible 
contexts. 

•  Identify  some  issues  of  concern  for  further  in-depth  investigation  (possibly  in  a 
more  quantitative  analysis). 

•  Define  and  use  models  to  investigate  and  analyse  specific  problem  issues. 

•  If  required,  exploit  results  of  models  and/or  knowledge  from  standard  science. 

Assessment: 

•  Design  possible  options  for  changes  to  the  system  of  interest. 

•  Do  some  ‘what  if  analysis  to  select  a  short  list  of  possible  options  for  further  study. 

•  Compare  options. 

Action: 

•  Offer  support  to  the  client  in  preparing  his  preferred  courses  of  action,  compare 
pros  and  cons. 


Figure  4  illustrates  how  each  phase  may  include  tasks  (and  methods)  that  are 
typical  of  other  phases,  in  a  varying  intensity  and  most  likely  requiring  a  varying 
amount  of  time.  Of  course,  at  the  global  study  level,  the  major  emphasis  will  be 
on  the  tasks  and  the  use  of  methods  that  are  typical  of  a  specific  phase  (i.e.  the 
diagonal  from  the  lower  left  comer  to  the  upper  right  corner  of  Figure  4),  but  at 
the  local  study  level  within  a  phase  there  may  be  a  similar  sequence  of  study 
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phases.  Consideration  of  the  other  phases  will  also  ensure  that  the  overall  study 
goal  is  kept  in  mind  while  working  at  the  specific  phase. 


time  through  study 
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-oriented  tasks 


Assessment 
-oriented  tasks 


Analysis 
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-oriented  tasks 


Appreciation 
■oriented  tasks 


Figure  4:  Each  Main  Phase  Requires,  in  a  Varying 
Intensity,  Tasks  that  are  Typical  of  All  Phases. 

In  general,  the  appreciation  phase  is  likely  to  be  relatively  longer  for  a 
judgement-based  OA  study  than  that  for  a  more  quantitative  one.  This  does  not 
necessarily  mean  that  overall  a  judgement-based  OA  study  will  take  longer 
than  a  quantitative  one;  it  is  just  that  the  balance  of  formulation  and  resolution 
may  differ.  As  a  fundamental  rule  for  the  initial  stage  of  the  study,  the  analyst 
will  be  trying  to  identify  what  the  real  question  is  and  for  this  reason,  much 
of  the  initial  work  will  concentrate  on  the  appreciation  phase  (Figure  4). 
For  this,  the  analyst  has  to  establish  a  mutual  understanding  with  the  client, 
his  backgrounds  and  value-sets.  The  influences  of  the  other  stakeholders  also 
need  to  be  catered  for  (see  Chapter  5). 

It  is  often  the  case  that  when  faced  with  a  ‘problem’  or  ‘mess’,  an  intermediate 
report  may  be  produced,  termed  a  ‘front  end’  or  ‘scoping’  study.  This  is  a 
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valuable  way  of  providing  an  initial  appreciation  of  the  issues  ( see  case  study 
on  this  page).  Agreement  of  a  common  picture  of  an  issue  is  a  sensible  and 
necessary  step  (part  of  the  appreciation  phase)  before  work  on  solutions, 
options  or  ways  ahead  is  taken  further. 

There  are  a  number  of  methodologies  available  that  can  support  a  ‘front  end’ 
problem  formulation  in  a  scoping  study  ( see  the  companion  analyst’s  volume). 
Experience  has  shown  that  a  single  technique  is  rarely  enough  for  a  typical 
study;  instead  methodologies  are  adapted  or  in  some  cases  a  multi¬ 
methodology  approach  ( see  case  study  on  Page  23)  is  taken.  The  client  should 
not  request,  let  alone  insist  upon  a  specific  methodology,  but  instead  clearly 
articulate  the  aims  and  objectives  of  the  study  and  work  in  partnership  with  the 
analyst  to  design  an  approach  best  suited  to  the  study’s  aims. 

Case  Study:  An  Initial  Study  to  Bound  Future  Work  - 
Future  Expeditionary  Operations 


In  NATO  a  capability  manager  was  asked  to  consider  the  future  capabilities 
that  might  be  needed  for  expeditionary  operations.  He  took  the  NATO  definition 
of  Expeditionary  Operations  as  his  starting  point;  however  he  found  it  vague 
and  all-encompassing  and  therefore  not  useful  to  direct  or  inform  capability 
requirements  analysis.  Help  was  needed  to  delimit  and  more  clearly  define 
expeditionary  operations  before  a  study  of  requirements  could  begin. 

Morphological  analysis  was  used  to  identify  common  understanding  about 
the  key  characteristics  for  expeditionary  operations  across  various 
communities  within  NATO.  It  was  ideal  as  a  tool  to  deal  with  the  multi¬ 
dimensional  and  non-quantifiable  nature  of  the  problem.  The  study 
resulted  in  a  more  detailed  description  of  expeditionary  operations 
that  could  be  used  as  the  basis  for  further  requirements  work. 
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Case  Study:  An  Example  of  Multi-Methodology - 
Maritime  Mine  Countermeasures 


A  study  conducted  for  a  Nation’s  Navy  on  'New  Operational  Concepts  for 
Maritime  Mine  Counter  Measures’  involved  addressing  a  multitude  of  aspects 
and  drawing  on  many  different  areas  of  expertise.  The  study  was  conducted 
using  a  multi-methodology  approach  in  a  series  of  facilitated  workshops: 

•  Methods  used  in  the  1st  work  package  were: 
scenario  development  and  analysis,  capability  analysis. 

•  Methods  used  in  the  2nd  parallel  work  package  were: 
technology  survey  and  assessment. 

•  Methods  used  in  the  3rd  subsequent  work  package  were: 
morphological  analysis  (for design),  multi-criteria  analysis  (forevaluation). 

•  Methods  used  in  the  4th  subsequent  work  package  were: 
additional  technological  analysis,  in-depth  multi-criteria  analysis. 

Facilitation  played  a  key  role  in  helping  study  participants  stay  focused  while 
addressing  the  complexity  of  the  problem.  Facilitation  also  helped  ensure 
that  participants  were  aware  of  the  particular  perspective  by  which  the 
current  project  stage  was  to  be  looked  upon.  The  facilitator  explained  current 
discussion  issue(s),  repeated  crucial  current  assumptions,  clarified  the 
current  activity’s  position  in  the  chain  of  activities  being  undertaken,  ensured 
that  conclusions  were  not  adopted  without  analysing  their  rationale, 
managed  expectations  and  managed  the  changing  views  regarding  ways 
of  structuring  and  using  the  base  material.  The  facilitator  worked 
closely  with  the  analyst  to  progress  the  study. 


Facilitated  workshops  ( see  case  study  on  this  page)  are  often-used  tools  in 
judgement-based  OA  to  provide  a  venue  for  stakeholders  to  meet  in  order  to 
inform  each  other,  to  agree  on  problem  formulations  and  to  create  courses  of 
action.  A  central  requirement  for  conducting  successful  workshops  is  an 
independent,  impartial  facilitator.  The  facilitator  should  observe  roles  and  role 
behaviour  and  needs  to  be  aware  of  the  power  and  politics  that  may  affect  the 
group.  It  is  therefore  noted  that  a  facilitator  is  likely  to  play  an  important  role 
by  providing,  for  example: 

•  Help  so  that  the  participants  can  work  together  and  focus  the 
discussion; 

•  Management  of  the  elicitation  process  in  an  unbiased  manner;  and 

•  Facilitation  so  that  workshops  are  not  dominated  by  individuals. 
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The  typical  conceptual  environment  in  which  judgement-based  methods  come 
into  play  is  shown  in  Figure  5. 
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Figure  5:  Judgement-Based  OA  and  the  Role  of  the  Code  of  Best  Practice. 
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Figure  5  is  itself  deliberately  displayed  as  a  diagram  including  relationships 
between  concepts  as  this  is  one  of  the  methods  that  analysts  use  to  portray 
(‘model’)  systems  under  study,  particularly  in  cases  like  this  where  a  process  or 
action  is  followed.  Identification  of  actions  and  related  stakeholders,  linkages, 
feed-back  loops  and  the  influences  of  each  element  provide  a  common  picture 
of  the  issues  involved.  Its  purpose  is  creating  clarity,  focus  and  enabling 
communication  and  debate.  While  the  diagram  is  at  first  glance  complicated, 
there  are  no  difficult  concepts  and  it  contains  all  the  necessary  elements  in  an 
easily  followed  form.  All  judgement-based  studies  will  follow  this  basic 
pattern  of  developing  a  model  (or  models),  manipulating  the  model  to  explore 
changes  and  then  proposing  options  and  means  to  compare  them. 

As  can  be  seen,  the  notion  of  a  CoBP  is  expressed  with  the  above  environment 
in  mind.  For  much  of  the  analytical  activity  the  client  is  either  specifically 
involved  at  a  node  (e.g.  ‘Decision  makers’  and  ‘Stakeholders’),  influences  or  is 
influenced  by  a  neighbouring  node.  Thus  the  client,  the  problematic  situation 
and  the  analyst  are  intimately  involved.  In  this  client-oriented  volume,  Chapters 
2  and  7  describe  the  nature  of  problematic  situations  and  the  requirements  of  the 
decision  makers;  Chapter  3,  how  the  study  issues  evolve  and  sometimes 
require  a  more  quantitative  analysis;  Chapter  4,  the  methods  used  and  their 
input;  Chapter  5,  the  role  of  the  analysts  and  clients  and  form  of  the  output;  and 
Chapter  6,  study  validation,  credibility  and  acceptance. 


In  summary,  despite  the  uniqueness  of  each  study,  there  are 
common  approaches  that  provide  a  methodical  way  to  conduct 
judgement-based  OA.  Considerable  emphasis  is  placed  on 
problem  definition  or  appreciation.  The  process  is  iterative  and 
involves  extensive  interaction  of  the  client  and  analyst  groups. 
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In  a  judgement-based  OA  study,  there  may  be  many  issues,  each  ‘owned’  by  a 
stakeholder.  The  number  of  individuals  and  interest  groups  could  therefore  be 
significant.  As  shown  in  Figure  6,  even  the  notion  of  ‘client’  has  several 
meanings  including  decision  makers,  customers,  sponsors  and  end  users.  Some 
of  the  clients  may  also  have  additional  roles  as  subject-matter  experts. 


Figure  6:  Individuals  and  Interest  Groups  Involved 
in  a  Judgement-Based  OA  Study. 


The  roles  and  responsibilities  of  each  of  the  individuals  involved  should  be 
defined.  This  is  a  full  list  of  roles,  and  in  some  studies  a  single  individual  may 
take  on  more  than  one  role.  For  instance,  any  of  the  individuals  involved  can  be 
a  stakeholder.  Similarly,  an  analyst  may  also  work  as  a  facilitator  in  a  study. 


26 


RTO-SAS-087 


WHAT  IS  EXPECTED  OF  ME,  THE  CLIENT? 


Potential  participants  in  a  judgement-based  OA  study  are: 

•  Stakeholder  -  someone  affected  by,  or  involved  with  making, 
the  decision. 

•  Clients: 

•  Sponsor  -  the  client  who  owns  the  study  (and  often  the 
‘problem’). 

•  End  users  -  the  clients  who  will  benefit  from  the  study’s 
outcomes. 

•  Customer  —  the  client  who  commissions  the  study  and  pays  the 
bill. 

•  Decision  makers  -  the  clients  with  influence  and  power  to 
decide  on  issues  and  effect  change. 

•  Others: 

•  Subject-Matter  Expert  -  someone  who  knows  about  the  domain 
under  study. 

•  Scrutineer  -  someone  who  independently  looks  at  the  study 
process  and  results. 

•  Facilitator  -  an  impartial  catalyst  to  draw  knowledge  from  others 
and  help  people  work  together. 

•  Analyst  -  someone  who  designs  and  conducts  a  study  and  knows 
what  methods  to  use. 

The  mix  of  those  involved  in  the  study  is  rich  and  a  balance  needs  to  be  struck 
between  the  divergent  exploratory  phase  where  the  issues  are  developed,  and 
the  convergent  phase  of  consensus  and  action  where  the  insights  from  the  study 
are  developed  (Figure  2).  As  can  be  inferred,  not  all  of  the  ‘client’  group  would 
be  involved  at  any  one  time.  The  observation  that  almost  half  of  the  roles 
mentioned  in  the  above  list  are  labelled  as  ‘clients’  brings  considerable 
responsibilities  for  that  group.  These  responsibilities  include: 

•  Adopting  ownership  of  the  study  and  its  results; 

•  Providing  insights  on  the  problematic  situation  (e.g.  current  perception 
of  the  issues,  of  key  individuals,  of  the  sources  of  expertise  that  the 
study  could  draw  on); 

•  Assisting  the  analyst  in  developing  and  evolving  the  study  design  as 
the  study  progresses; 
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•  Approving  the  study  design  and  changes  to  it; 

•  Providing,  or  helping  ensure  access  to,  subject-matter  experts  and 
some  of  the  information,  data,  and  documents  needed  during  the 
course  of  the  study;  and 

•  Participating  in  and  encouraging  (possibly  busy)  other  individuals  to 
contribute  to  the  study  (e.g.  by  issuing  invitations  to  participate  in 
workshop  sessions,  by  providing  an  introduction  for  the  analyst  to 
other  stakeholders,  by  building  a  high  level  of  visibility  and  support 
for  the  study  within  the  client  community). 

Without  intimate  involvement  of  the  client,  particularly  for  access  to  observed 
data,  any  study  is  likely  to  be  both  shallow  and  narrow  and  likely  to  require 
rework  when  exposed  to  the  decision  makers.  With  such  material,  however, 
both  the  quality  of  the  study  and  the  trust  in  the  results  increases  ( see  case 
study  on  this  page). 

Case  Study:  The  Importance  of  Access  to  Data  - 
Analysis  of  ‘White’  Shipping 


In  the  development  of  NATO  maritime  capability,  user  requirements  for 
monitoring  ‘white’  shipping  were  not  well  understood.  To  improve 
understanding,  a  cognitive  task  analysis  was  conducted  using 
critical  incident  inquiry  to  identify  the  specific  cues  that 
alert  operators  to  suspicious  behaviour. 

Concept  mapping  was  employed  to  record  and  analyse  operator  strategies 
and  reasoning  as  they  worked  with  systems  to  complete  tasks.  Performance 
and  standard  work  load  measures  were  also  collected  and  combined 
with  the  concept  maps  to  provide  an  overall  assessment. 

Expert  and  well-calibrated  observational  data  was  required  for  reasonable 
interpretation  of  the  way  operators  use  strategies.  While  some  data 
collectors/observers  were  comfortable  with  the  tools  and  methods  being 
used,  several  were  only  available  for  two  days  before  the  execution  to  train 
and  exercise.  This  caused  some  difficulty  in  the  analysis,  and  necessitated  a 
much  greater  amount  of  post  experiment  analysis.  Even  with  a  standard 
method  for  recording  observational  data,  collectors  need  to  have  appropriate 
time  and  training  to  be  familiar  with  the  relevant  methods  and  theoretical 
implications  associated  with  behaviours  being  observed. 


The  final  point  to  be  made  is  that  while  the  analyst  has  the  lead  in  the  design  of 
the  study,  the  client  will  strongly  influence  how  the  outputs  should  be  presented. 
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The  analyst  can  provide  advice  on  this,  of  course,  but  the  client  should  be  more 
attuned  to  the  use  that  would  be  made  of  the  product,  and  the  decision-making 
process.  Whether  for  a  more  quantitative  or  more  judgement-based  OA  study, 
it  is  probable,  if  not  certain,  that  discussion  material  will  be  presented  to  the 
decision  makers.  For  a  more  quantitative  study  the  raw  data  will  probably  be 
reasonably  clear-cut  even  though  there  may  still  be  a  contestable  decision 
(e.g.  the  reliability  versus  cost  issue  described  in  Chapter  1).  For  a  judgement- 
based  OA  study,  the  client  and  analysts  should  consider  two  issues:  the  form  of 
the  actual  results  and  their  presentation.  If  done  badly,  either  of  these  aspects 
could  render  an  otherwise  excellent  study  worthless.  In  addition,  the  relationship 
between  the  analyst  and  the  client  can  be  damaged. 

The  text  box  on  this  page  shows  a  generic  acquisition  study  that  will  be  used  to 
illustrate  many  of  the  issues  involved  in  the  types  of  decision  making  using 
multiple  criteria  and  likely  to  be  found  in  judgement -based  OA  studies. 

A  Generic  Acquisition  Study 

Study  Purpose: 

To  compare  three  hardware  options  (A,  B  and  C)  for  a  capability  acquisition. 

Formulation: 

Agree  that  ‘capability’  is  a  combination  of  criteria,  e.g.  ‘lethality’,  ‘mobility’, 

‘survivability’  and  ‘sustainability. 

Method: 

Find  ways  to  measure  or  otherwise  provide  insight  on  these  terms  for  each 

of  the  options. 

Output: 

A  compilation  of  the  pros  and  cons  of  the  three  options. 


An  example  of  presenting  the  data  for  this  generic  acquisition  study  is 
shown  in  Figure  7  (and  its  explanation  in  the  text  that  follows).  This  illustrates 
several  important  learning  aspects  based  on  the  notions  of  criteria,  numbers  and 
colours.  It  is  worth  discussing  these  aspects  in  detail  as  they  provide  several 
informative  examples  for  the  use  and  applicability  of  judgement-based  OA. 

Two  ways  of  presenting  the  data  are  shown  in  the  same  table  of  Figure  7. 
The  first  is  primarily  based  on  a  pictorial  representation  including  colour  based 
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on  the  use  of  underlying  threshold  values.  The  second  includes  quantitative 
tabulated  data.  The  results  can  be  interpreted  in  terms  of  the  observation  that 
Option  C  appears  to  have  lower  capability,  though  neither  Option  A  nor  Option 
B  fulfils  all  the  criteria. 

The  point  being  made  with  this  example  is  that  the  final  part  of  the  judgement 
has  been  left  to  the  decision  maker  -  it  is  expected  of  him,  in  this  particular 
example,  to  balance  the  relative  strengths  and  weaknesses  of  the  three  options. 
Thus  the  nature  of  a  judgement-based  OA  approach  means  that  it  should  be 
presented  in  a  way  that  supports  decision  making  and  that  judgement  and 
choice  can  still  be  applied  by  the  decision  makers  and  the  raw  results  do  not 
impose  a  strait  jacket.  The  way  of  presenting  results  is  primarily  the  analyst’s 
responsibility,  but  requires  involvement  of  the  client;  the  client  should  therefore 
be  prepared  to  be  actually  involved  in  the  presentation  and  interpretation  of  the 
results. 


In  summary,  the  client  is  one  of  several  stakeholders. 
The  client  should  be  actively  involved  in  all  stages  of  the 
judgement-based  OA  study,  including  problem  definition,  study 
planning  and  execution,  data  gathering,  and  design  and  form 
of  the  study  outputs. 
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Option  A  Option  B  Option  C 


Criterion  1 

90% 

95% 

90% 

Criterion  2 

90% 

85% 

80% 

Criterion  3 

75% 

95% 

Criterion  4 

95% 

90% 

75% 

Figure  7:  ‘Traffic  Light’  to  Compare  Options  Which  Allows 
Freedom  of  Choice  by  the  Decision  Makers. 


The  use  of  four  separate  criteria  allows  freedom  of  action  to  the  decision  maker  by 
leaving  the  final  part  of  the  judgement  to  the  decision  maker  -  it  is  expected  of  him, 
in  this  particular  example,  to  balance  the  relative  strengths  and  weaknesses  of 
the  three  options  A,  B  and  C.  To  assist  in  this,  analytical  techniques  have  been 
developed  that  allow  consensus  on  the  relative  importance  of  criteria  to  be 
converted  to  weighting  factors  and  these  can  be  used  to  identify  a  single  value  for 
each  of  the  options.  Sensitivity  analysis  would  allow  inspection  of  how  susceptible 
these  figures  are  to  changes  in  relative  weighting,  which  may  in  fact  be  considerable. 
It  is  critical  that  the  decision  maker  is  involved  in  this  process  and  is  left  with  room 
for  manoeuvre.  There  would  however  be  little  room  for  manoeuvre  if  the  material 
was  merely  presented  in  the  form  of  a  (equally)  weighted  average  of  the  four 
criteria:  option  A  88%,  B  91%  and  C  73%.  Thus  the  nature  of  a  judgement-based 
OA  approach  means  that  it  should  be  presented  in  a  way  that  supports  decision 
making  and  that  judgement  and  choice  can  still  be  applied  by  the  decision  makers 
and  the  raw  results  do  not  impose  a  strait  jacket. 
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The  use  of  numbers  is  a  double-edged  sword.  Quantitative  material  can  provide 
compelling  evidence  and  can  help  in  the  scrutiny  process.  There  may  even  be  a 
perception  that  quantitative  results  take  precedence  over  all  other  forms  of  input. 
However,  the  methods  used  in  the  study  would  ideally  be  based  on  an  agreed  plan 
of  work  and  by  necessity  will  include  the  use  of  approximations  and  subjective 
grouping  of  disparate  elements  to  reach  these  numbers.  It  is  also  by  no  means 
certain  that  the  criteria  are  even  mathematically  discrete.  In  such  cases,  the  use  of 
numbers  is  to  inform  the  decision-making  process  rather  than  to  provide  definitive 
material.  In  the  paragraph  above,  it  was  noted  that  a  single  figure  could  be 
calculated  for  each  option,  but  then  the  richness  in  the  study  has  been  lost  by  this 
approach  and  the  executive  has  not  been  given  a  complete  picture  on  which  to 
make  a  decision.  There  is  a  temptation  to  take  the  actual  numbers  too  literally  and 
prefer  Option  B  over  Option  A  even  though  the  numbers  should  only  be  taken  as 
indicative.  The  decision  maker  also  loses  the  ability  to  prefer  Option  C  which  may 
be  preferred  when  other  factors  are  taken  into  account. 

The  third  issue,  colour,  is  possibly  the  most  complex.  As  noted  in  Tables  1  and  3 
emotion  is  an  important  part  of  judgement-based  OA  and  such  ‘traffic  light  analysis’ 
is  likely  to  play  a  part  in  perceptions.  An  obvious  question  of  Figure  7  is  the  basis  of 
the  colour  coding  -  at  what  threshold  value  does  an  option  achieve  a  green  rating? 
This  is  something  that  should  have  been  agreed  between  the  analysts  and  client 
before  the  material  was  submitted  to  the  decision-making  process  but  should  be 
transparent.  For  instance  it  could  be  based  on  a  requirements  analysis.  The  use  of 
colour,  however,  does  have  benefits.  For  instance,  options  A  and  B  both  have 
three  ‘greens’  and  one  ‘yellow’.  This  presents  the  issue  well  to  the  decision  maker  - 
where  do  they  want  to  take  the  ‘yellow’?  Similarly,  the  ‘red’  for  option  C  flags  a 
critical  issue  for  further  discussion. 
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The  inherent  uncertainty  of  a  messy  situation  leads  the  client  for  judgement- 
based  OA  to  what  are  perhaps  his  most  pressing  concerns  -  the  need  for 
validity,  credibility  and  acceptance  ( see  key  aspect  3  in  Chapter  1). 

The  analyst  should  be  considered  the  expert  in  design,  development  and 
implementation  of  the  study  plan  and  methods.  He  will  be  guided  by  established 
practice,  previous  work  and  experience  and  by  general  skills  in  the  area.  One  of 
these  skills  derives  from  his  knowledge  of  how  to  ensure  that  a  sound  process 
has  been  followed  in  two  respects: 

•  Was  the  approach  taken  appropriate  for  the  maturity  of  the  issue? 
(was  the  right  study  done?). 

•  Was  ‘due  diligence’  employed  in  carrying  out  the  study?  (was  the 
study  done  right?). 

Was  the  Right  Study  Done? 

The  analyst  community  recognises  two  seemingly  distinct  categories  of 
problematic  situation  where  a  predominantly  judgement-based  study  is 
appropriate:  problems  and  messes  (Table  1).  When  considering  the  defence 
sector,  as  a  general  rule  it  can  be  supposed  that  the  position  in  the  capability 
development  cycle  will  impact  on  the  type  of  problematic  situation  encountered 
and  thus  the  style  of  study,  as  follows: 

•  Needs  or  concept  phase  -  ‘Mess’  is  most  appropriate  as  there  should 
be  a  need  for  a  high-level  examination  of  the  issues  from  which 
concepts  can  be  developed  in  the  form  of  capability  options.  Typical 
examples  of  a  needs  phase  study  include  policy  or  concept  development, 
gap  analysis  and  broad  options  for  force  structure  change. 

•  Requirements  phase  -  A  ‘mess’  or  a  ‘problem’  is  most  appropriate  as 
the  capability  options  begin  to  take  shape  and  more  definition  and 
specification  is  required.  Typical  examples  of  requirements  phase 
study  include  option  development  and  comparison,  possibly  leading 
to  the  start  of  an  acquisition. 
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•  Acquisition  phase  -  Depending  on  the  degree  of  certainty  of  the 
issues,  a  ‘problem’  or,  less  likely,  a  ‘puzzle’  approach  would  be 
appropriate.  Typical  examples  where  judgement-based  OA  is  involved 
include  comparison  of  operational  concepts  for  different  short-listed 
equipment  options. 

•  In-service  -  Any  fit-for-purpose  approach  is  appropriate,  particularly 
if  there  are  time  or  other  constraints  such  as  in  operations. 

Figure  8  suggests  a  procedure  that  allows  examination  of  problematic 
situations  where  there  is  initial  uncertainty  of  how  to  proceed.  Using  the 
previously  discussed  material  it  is  straightforward  to  separate  a  ‘puzzle’  from  a 
situation  where  many  of  the  elements  in  Table  1  in  Chapter  2  were  ticked. 
At  this  stage,  it  is  useful  to  consider  the  situation  as  a  ‘perceived  mess’  but  note 
that  with  examination  by  the  analyst  and  client  some  cases  may  be  turned  into 
‘problems’.  In  these  cases  there  will  be  a  commitment  to  an  agreed  study  plan 
developed  jointly  by  the  client  and  analyst.  This  conversion  of  a  ‘perceived 
mess’  to  a  ‘problem’  is  often  considered  as  an  important  aim  of  judgement- 
based  OA  and  is  often  referred  to  as  ‘problem  structuring’.  Typical  elements  of 
concern  of  any  study  of  a  problematic  situation  are: 

•  Setting  initial  boundaries  in  physical,  organisational,  social,  time- 
related,  etc.,  terms  as  they  could  affect  the  analysis  approach; 

•  The  nature  of  the  options  to  choose  among,  and  the  ability  of  (semi-)- 
quantitatively  evaluating  and  comparing  them; 

•  The  type(s)  of  model  to  be  constructed  and  used,  and  the  desired 
nature  of  the  results  and  the  outcome  of  the  study  as  a  whole; 

•  The  participants  in  the  study,  the  specific  types  of  expertise  needed, 
and  the  information  each  individual  will  bring  in  through  their 
expertise;  and 

•  The  study  design  needed  to  achieve  results. 
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Figure  8:  Methodology  for  Coping  with  Problematic  Situations  and  Their 
Nature  Where  There  is  Initially  Uncertainty  of  How  to  Proceed. 


Other  cases,  however,  will  be  better  treated  as  messes.  Some  analysts  believe 
that  they  are  ‘managed,  not  solved’.  This  leads  to  a  different  analytical  course 
of  action,  for  instance: 

•  A  better  understanding  of  aspects  of  the  problem  and  the  factors  that 
influence  it; 

•  Identification  of  problematic  aspects  that  should  be  addressed  and 
improved; 

•  A  view  of  any  boundaries  to  be  set  initially; 

•  The  nature  of  the  anticipated  outcome  of  the  study; 

•  The  participants  in  the  study,  the  specific  types  of  expertise  needed, 
and  the  information  each  individual  will  bring  in  through  their 
expertise;  and 

•  A  general  and  flexible  outline  of  the  study  design  to  achieve  results. 
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The  key  message  thus  is  fitness  for  purpose,  or  to  put  it  in  another  way  ‘the 

right  study  is  done’. 

Was  the  study  done  right? 

There  are  a  number  of  possible  challenges  to  the  validity  of  a  study. 

Validity  is  a  multi-faceted  concept.  Its  aspects  include: 

•  Objectivity  (the  quality  of  being  based  on  rules  grounded  in  theory  or 
established  practice  and  characterised  by  recorded  argumentation  and 
rationale  and  following  an  agreed  and  sound  process  accepted  by  all 
involved); 

•  Rigour  (quality  achieved  through  strict  enforcement  of  rules  and 
requirements); 

•  Repeatability  (the  quality  of  an  outcome  to  occur  again,  given  same 
starting  conditions  -  it  might  be  noted  that  this  is  a  worthy  goal  but 
it  may  not  occur  as  similarity,  particularly  when  individuals  or 
organisations  are  involved,  can  not  be  guaranteed); 

•  Auditability  and  transparency  (the  critical  elements  that  allow 
scrutiny  through  validation  and  verification  of  the  study  approach  and 
findings); 

•  Independence  (the  proactive  steps  taken  to  ensure  lack  of  bias, 
whether  intentional  or  unintentional); 

•  Grounding  and  consistency  (a  validation  mechanism  where  studies 
are  based  on  theory  or  established  methods  and  can  be  checked 
against  real  or  calculated  data); 

•  Understand-ability  (the  removal  of  barriers  to  learning  and 
involvement  in  the  study  achieved  through  the  use  of  terms  familiar 
to  the  clients  and  through  explanation  of  the  methods); 

•  Explanatory  power  (the  depth  and  breadth  of  explanation  of  the 
system  properties  and  behaviours  inherent  to  the  problematic  situation 
as  afforded  by  the  OA  method  and  its  results); 

•  Completeness  (demonstration  that  the  methods  adopted  for  the  study 
address  all  aspects  of  a  problematic  situation,  and  if  not,  the  limitations 
that  are  imposed); 
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•  Robustness  under  uncertainty  (the  accommodation  of  incompleteness 
and  variability  and  its  implications);  and 

•  Clear  separation  of  data  from  the  method  which  uses  it  (the  quality  of 
ensuring  that  data  availability  does  not  drive  method  selection  and 
vice  versa). 

There  is  a  need  for  rigour  in  the  absence  of  verifiable  mathematical  methods. 
Thus  a  sound  procedure,  based  on  the  CoBP,  is  followed  to  ensure  that  there  is 
an  audit  trail  showing  how  the  issues  have  been  addressed.  The  aim  is  to 
achieve,  from  use  of  the  CoBP,  sufficient  rigour  to  render  the  entire  process 
objective,  even  though  individual  elements  of  a  judgement-based  OA  study 
may  require  subjective  inputs.  Put  simply,  the  analyst  and  client  have  to  work 
harder  in  a  judgement-based  OA  study  to  develop  a  convincing  case  than  they 
would  if  the  results  fell  naturally  from  a  mathematical  process. 

The  professional  analyst  will  have  many  attributes  and  skills  to  bring  to  the 
problematic  situation  (Table  5)  which  will  enhance  the  likelihood  that  validity, 
credibility  and  acceptance  will  be  manifest.  He  will  be  responsible  for 
assembling  an  analytical  team  comprising  the  appropriate  skill  sets  or  access  to 
computer-based  or  other  resources.  In  a  similar  way  to  the  client  roles 
expressed  in  Chapter  5,  the  other  defined  roles  of  the  analyst  are: 

•  Help  structure  and  define  the  nature  of  the  problematic  situation; 

•  Design  the  stages  of  the  process  used  within  the  study; 

•  Suggest  scientific  ways  to  investigate  and  model  the  problematic 
situation  and  suggest  the  methodology  and  the  method(s)  to  be  used, 
the  workshops  (if  any)  to  be  held,  and  ways  to  report  and  interpret 
the  study's  outcomes; 

•  Provide  facilitated  modelling  (generally  through  the  use  of  workshops; 
including  an  integrated  process  of  data  gathering  and  model  creation 
and  analysis); 

•  Help  develop  agreed-upon  outputs  (e.g.  better  understanding  of  the 
problematic  situation,  identification  of  desirable  and  politically 
feasible  options  for  improving  the  problematic  situation); 

•  Report  and  interpret  the  study  process  (including  participants),  results 
and  outcomes; 
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•  Generally,  provide  a  rational  and  logical  analysis  that  aids  in 
managing  the  complexity  of  the  problematic  situation,  aids  in 
recognising  and  managing  uncertainty  and  risk,  and  adequately 
meets  the  overall  study  goals;  and 

•  Remain  thoroughly  devoted  to  providing  a  credible,  rigorous  analysis 
and  to  ethical  principles. 

Table  5:  Attributes  of  a  Judgement-Based  OA  Analyst. 

A  competent  judgement-based  OA  analyst: 

•  Is  interested  in  practical  solutions. 

•  Can  take  a  broad  holistic  view  of  the  issue  under  study. 

•  Possesses  a  high  degree  of  initiative,  energy,  and  maturity. 

•  Is  a  ‘self-starter’  who  can  work  well  individually  and  in  a  team. 

•  Has  a  high  degree  of  general  intelligence  and  enthusiasm  for  the  work. 

•  Knows  relevant  judgement-based  methods  and  their  merits  (pros  and  cons). 

•  Enjoys  drawing  knowledge  from  other  disciplines. 

•  Has  the  right  personality  so  as  to  be  able  to  gain  client  confidence  and 
effectively  communicate  study  results  to  decision  makers. 

•  Is  devoted  to  scientific  rigour  and  ethical  principles. 

•  Has  adopted  and  shows  a  fundamentally  helpful  attitude. 


The  correctness  of  a  judgement-based  OA  study  is  hard  to  define  and  for  this 
reason  ‘due  diligence’  guidance  has  been  implemented  by  the  Task  Group  as  a 
CoBP.  Similarly,  it  is  difficult  to  define  a  specific  study  pattern  of  work. 
An  approach  based  on  principles  or  values  is  preferred  over  a  rigid  manual 
where  every  step  has  to  be  completed  before  the  next  can  be  started. 

The  primary  dimensions  of  validity  are  objectivity  and  rigour  ( see  Page  36); 
credibility  and  acceptance  are  not  distinct  qualities  but  are  rather  derived  from 
validity  and  reinforced  by  the  skills  and  expertise  of  the  analyst. 

Credibility  is  achieved  when  expertise  is  delivered  with  trustworthiness. 
The  level  of  credibility  a  study  method  may  have  will  depend  on  the  amount  of 
expertise  available  to  apply  it,  and  the  perceptions  of  the  client  (and  other 
stakeholders)  of  that  expertise.  Indeed,  there  are  both  subjective  and  objective 
components  to  credibility.  In  particular,  the  trustworthiness  of  an  analyst  will 
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itself  be  subjective  and  personal  to  clients.  In  order  to  gain  credibility  for  those 
methods  which  have  been  validated  through  consideration  of  the  above  factors, 
it  is  necessary  to  also  consider  the  acceptance  of  such  models  from  the  decision 
maker  community. 

Acceptance  is  a  concept  which  is  present  throughout  the  study  process. 
It  starts  at  the  point  where  the  problematic  situation  is  raised  and  is  critical 
through  to  the  end  of  the  process,  where  the  analysis  will  be  exploited  by 
clients.  Communication  between  the  analyst  and  the  clients  is  critical 
throughout  all  aspects  of  the  study  process;  often  lack  of  communication  with 
stakeholders  can  be  missed  until  the  very  end  of  a  study  and  so  damage  its 
acceptance.  Acceptance  can  be  gained  through  transparency  and  communication 
of  the  modelling  process.  This  will  help  the  analyst  gain  the  trust  and 
confidence  of  the  decision  makers  in  the  method,  and  more  importantly,  the 
study  outcomes  so  that  these  can  be  of  most  value. 

The  confidence  a  client  has  in  an  analyst  can  also  be  increased  through 
adequate  preparation.  For  example,  training  an  analyst  improves  their  own 
confidence  which  will  then  be  more  easily  portrayed  to  clients. 

Credibility  and  acceptance  will  be  further  reinforced  by  independent  scrutiny. 


In  summary,  the  analyst  will  work  with  the  client  to  ensure 
that  an  appropriate  degree  of  validation  has  been  applied  to  a 
judgement-based  OA  study.  The  analyst’s  skills  and  expertise 
are  crucial  in  this  respect.  A  study’s  validity  and  the  (perceived) 
trustworthiness  of  an  analyst’s  expertise  will  to  a  large  extent 
determine  that  study’s  credibility,  and,  together  with 
transparency  and  good  communication  throughout  the  study 
process,  the  acceptance  of  study  outcomes  by  decision 
makers. 
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A  judgement-based  OA  study  will  likely  be  only  one  input  into  a  decision¬ 
making  process.  In  contrast  to  the  added  value  described  in  Chapter  3,  a  set  of 
threats  to  a  successful  and  valid  judgement-based  OA  study  can  be  codified. 
These  threats  to  judgement-based  OA,  can  be  intentional  or  unconsciously 
directed  by  individuals’  decision-making  preferences  or  backgrounds.  They 
could  include: 

•  Threat  1:  Disagreement  with  a  specific  part  of  the  study  leading  to  a 
dismissal  of  the  rest  of  the  material. 

•  Threat  2:  Treating  judgement -based  OA  study  results  with  too  much 
certainty  (e.g.  as  a  prediction). 

•  Threat  3:  Too  rapid  a  progression  from  an  ill-formed  concern  to  a 
rigid  plan  for  change  (e.g.  an  acquisition). 

•  Threat  4:  Biased  preference  of  some  forms  of  evidence  (e.g. 
quantitative  sources). 

•  Threat  5:  selective  interpretation  to  support  a  specific  argument. 

Some  of  the  threats  (e.g.  1  and  5)  hold  for  OA  in  general,  but  may  be 
particularly  likely  to  occur  with  judgement-based  OA.  Some  of  these  threats 
will  occur  no  matter  the  situation  but  the  analyst  (and  where  appropriate  the 
client)  can  provide  convincing  arguments  to  address  and  mitigate  most  of  the 
items  on  the  list.  There  is  probably  little  that  can  be  done  about  Threat  5, 
except  perhaps  making  different  interpretations  of  parts  of  the  model  (e.g.  in 
the  form  of  a  visual  map  of  relevant  concepts)  so  that  they  are  represented  in 
the  results.  The  adoption  of  the  principles  of  the  CoBP  will  limit  the  impact  of 
unhelpful  criticisms  that  may  occur  in  a  real  study. 

Table  6  shows  how  a  hypothetical  situation  relating  to  an  acquisition  based  on 
military  effectiveness  may  be  challenged  by  these  threats.  Shown  are  mitigation 
strategies  to  counter  the  intent  of  Threat  1  and  also  to  address  legitimate 
analytical  scrutiny.  It  might  be  noted  that  the  example  in  Table  6  contains  far 
more  technical  detail  than  appears  in  the  remainder  of  this  volume.  This  is 
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introduced  for  illustrative  purposes  only  but  shows  some  of  the  issues  that  are 
likely  to  be  found  when  judgement -based  studies  are  done.  It  is  only  one  way 
in  which  such  a  study  may  be  conducted  and  is  based  on  the  generic  study  box 
in  Chapter  5.  The  companion  analyst  volume  contains  more  detail  of  such 
issues. 


Table  6:  Possible  Challenges  to  an  Acquisition-Focused 
Judgement-Based  OA  Study  (Example)  and  Their  Mitigation. 


Issue 

Mitigation  by  Analytical 
Approach 

Formal  Mitigation 

Identification  of 
Issues 

Exploratory  front-end 
(scoping)  analysis  for 
problem  definition  in  the 
appreciation  phase. 

Terms  of  reference  agreed 
by  the  customer. 

Definition  of 

Military 

Effectiveness 

Military  effectiveness  is 
represented  as  a  combination 
of,  for  example,  lethality, 
mobility,  sustainability  and 
survivability. 

Use  previous  studies  as 
support  and  gain  agreement 
of  the  client  group. 

Measurement 
of  Terms 
(e.g.  lethality) 

Measure  loss  exchanges 
ratios  through  war-gaming. 

Use  previous  studies  as 
support  and  gain  agreement 
of  the  client  group. 

Weightings 

Hold  workshops  with 
stakeholders. 

Good  practice  in  facilitation 
using  subject-matter  experts 
where  all  opinions  are  heard 
with  no  domination  by 
individuals. 

Vignettes  for 
Wargames 
(e.g.  for  lethality) 

Identify  a  range  of  fit  for 
purpose  vignettes  consistent 
with  any  prescribed  scenarios. 

Client  group  to  endorse  that 
they  are  suitable. 

Sample  Size/ 
Statistics 

Emphasise  quality  of  inputs 
rather  than  quantity  and 
include  material  derived  from 
different  sources. 

Client  group  to  provide 
quality  subject-matter 
experts.  Validate  through 
comparison  of  several 
different  information  sources. 

Data 

Use  a  combination  of 
literature,  commercial  and 
client  supplied  data  (informed 
opinion). 

Client  group  to  endorse 
material.  Include  sensitivity 
analysis. 
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Also  to  be  noted  in  Table  6  is  the  frequent  observation  of  required  agreement 
between  the  client  and  the  analyst,  which  may  be  informal  (though  recorded) 
or  embodied  in  the  terms  of  reference  which  form  part  of  the  study  plan. 

Threats  2  to  4  require  dialogue  between  the  analyst  and  members  of  the 
client  group.  The  material  in  this  volume  lays  the  foundations  for  a  robust 
defence  against  these  threats.  To  some  extent  the  position  taken  here 
involves  two  conflicting  elements: 

•  Certainty  that  a  good  process  has  been  followed;  and 

•  Uncertainty  in  the  results  and  outputs. 

Reconciling  the  two  might  seem  counterintuitive  but  that  is  the  purpose  of  the 
required  dialogue  between  the  client  group  and  the  analyst.  As  described  in 
Chapter  1,  the  three  key  aspects  of  judgement -based  OA  drive  the  study  and 
should  shape  the  use  that  is  made  of  any  study,  such  as  to  support  advocacy. 
This  debate  may  also  be  between  the  customer  and  the  sponsor  (see  Chapter  5). 
The  notion  of  a  ‘mess’  where  the  resultant  identification  of  management 
trajectories  and  progress  indicators  rather  than  a  project  approach  may  be  alien 
to  some  decision  makers,  particularly  those  involved  with  short  time  scales. 

Additionally,  there  is  a  need  for  expectation  management  where  the  threats  to 
the  study  are  caused  by  a  fundamental  difference  in  decision  making  and 
management.  Those  users  that  might  want  clear  direction  on  the  next  stage  may 
well  be  disappointed  as  the  conclusion  may  be  that  (for  instance)  a  firepower 
approach  has  more  merit  than  one  based  on  protection.  It  is  quite  possible  that 
the  study  will  end  at  this  point  with  a  recommendation  for  a  follow-on  study  to 
be  done,  e.g.  to  address  the  possible  options  for  firepower.  Nevertheless,  a  well- 
founded  decision  will  have  been  made,  and  critically,  the  basis  will  have  been 
set  for  the  criteria  and  approach  to  be  taken. 

Finally,  the  analyst  and  client  may  be  pushed  to  use  the  study  for  predicting 
what  is  going  to  happen  rather  than  forecasting  what  could  happen,  which 
again  confuses  the  role  of  this  sort  of  analysis. 

For  a  study  as  a  whole  there  will  be  a  clear  audit  trail  that  will  guide  from 
appreciation  of  the  issue  through  to  analysis  and  definition  of  ways  ahead. 
In  the  capability  development  process,  for  example,  this  will  mean  from  the 
initial  needs  phase  through  to  actual  selection  of  a  preferred  option. 
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The  CoBP  is  specifically  directed  to  counter  Threat  4,  as  implicit  to  any 
judgement-based  study  is  the  notion  that  while  quantitative  data  may  be  used  it 
will  be  used  in  specific  cases  to  test  an  idea,  or  it  might  be  applicable  to  a  small 
part  of  the  landscape.  If  the  issue  was  a  true  ‘puzzle’  then  the  appropriate 
mathematics-based  techniques  would  have  been  used.  For  ‘problems’  and 
‘messes’  it  is  more  likely  that  a  mix  of  quantitative  and  qualitative  data  will  be 
used  and  the  skill  is  in  melding  it  sensibly. 


In  summary,  judgement-based  OA  is  a  key  scientific  activity 
that  will  add  value  for  clients  involved  in  decision  making, 
allowing  decisions  to  be  made  with  less  risk.  However, 
potential  threats  to  a  judgement-based  OA  study  exist  and 
need  to  be  managed  through  the  application  of  the  principles 
of  the  CoBP. 
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